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ABSTRACT 

The study aims at determining the impact of learning driven constructs on Perceived Higher 
Order Cognitive Skills (HOCS) improvement when using multimedia and text materials. Perceived 
HOCS improvement is the attainment of HOCS based on the students' perceptions. The research 
experiment undertaken using a case study was conducted on 223 students split into two groups 
who used multimedia and text book instructional methods respectively, to determine the impact of 
exposure to technology on HOCS improvement of such skills as: decision-making, problem¬ 
solving, critical thinking, analysis, synthesis, interpretation. The data collected was analyzed 
using the independent sample t-test to examine the effects of Learning Driven constructs of 
multimedia and textbook on students perceived HOCS. The findings from the study suggest that 
multimedia materials had no measurable effect on HOCS as compared to the traditional text book 
approach. The study therefore concludes that there was no significant difference in the students' 
attitudes, learning interest, learned from others, self reported learning, and HOCS of decision 
making, problem solving, critical thinking, and other skills when multimedia and text book 
materials were used. 

Keywords: Perceived Higher Order Cognitive Skills, Learning Driven Constructs, Multimedia, 
Case study, Instructional methods. 


1. INTRODUCTION 

Higher Order Cognitive Skills (HOCS) are skills that go beyond basic comprehension of a 
problem or concept, and relate to the perception that an individual has acquired skills to make a 
decision under various conditions of uncertainty and time limit (Mbarika et at., 2010; Lou ef a/., 2008; 
Bradley et at., 2007a, b; Zoller et ai, 2002; Zoller, 1993; 1999). HOCS implies an improved 
learned ability to identify, integrate, evaluate, and inter-relate concepts within a given problem 
domain, thereby making the appropriate decisions to solve a problem. HOCS include skills of 
decision-making, problem-solving, critical thinking, analysis, synthesis, interpretation, among 
others. HOCS are essential if students are to apply knowledge acquired in class in novel and real 
life situations. Several authors imply that the traditional learning approach, a structured clear 
method that allows face to face interaction of students and their lecturers is not compatible with 
promotion and improvement of students' HOCS (Mbarika et al., 2010; Zoller & Pushkin, 2007; 
Bradley et al., 2007; Zoller, 1993; Juwah, 2003), therefore do not favor development of problem 
solving, critical thinking and decision making skills needed by the employer (Ball & Garton, 2005; 
Mbarika, 2003). 
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Undergraduate education is often criticized for failing to improve HOCS for problem solving and 
decision making in science courses like Computer Science, IT and Engineering (Mbarika ef a/., 
2010; Bagarukayo et al., 2007; Bradley ef a/., 2007a, b; Broussard et al., 2007; Zoller & Pushkin, 
2007). For example, students taught theory without opening a computer to show them how 
computer parts are a installed, upgraded or repaired, may not give them good grounding of the 
concept. Students need exposure to troubleshooting in a simulated real work environment to 
obtain problem solving, critical thinking and decision making skills. Although educators typically 
understand the learning objectives of their courses, the reality is that these objectives are rarely 
fully attained (Mbarika et al., 2010; Bradley et al., 2007). Introductory courses have been taught 
using the traditional learning approach, during which the instructor lectures on these technical 
topics. It is a common complaint for educators that science students did not have HOCS to take 
upper-level classes and that they often have to redo topics that were supposed to be attempted in 
the introductory classes (Bagarukayo & Mbarika, 2008; Mbarika et al., 2003a). This is reflected in 
many science graduates e.g. engineering, chemistry and IT, today. The complexity for people to 
deal with decision making situations that require HOCS has been identified (Mbarika et al., 2010; 
Bradley et al., 2007). The decision making process involves challenges such as selecting among 
alternatives that could have slight differences but with multi-million dollar implications. Therefore 
making a decision among the possible alternatives or a combination of alternatives requires that 
the decision makers apply HOCS during the selection process. 

As a result, there has been an emphasis over time on preparing students to graduate as 
proficient problem solvers, decision makers, and team players with good analytical skills, 
reasoning, problem identification, so as to enhance their performance in the real life work 
environment (Bradley et al., 2007). The relevance for graduates' HOCS for decision making, 
above average problem solvers, team players, in the industry to enable companies to make 
business profit has also been highlighted (Guzdial & Soloway, 2002; Rieley & Crossley, 2000; 
King, 2000). HOCS prepare students for today's workforce because graduates gain skills for 
problem identification, analysis, figuring out alternatives, evaluating alternatives, reasoning, 
synthesis, interpretation, question-asking problem solving, decision making, system and critical 
thinking, among others. Since research has shown that the current graduates lack these skills, 
there is a need for new teaching and learning methods, which impact on HOCS to prepare future 
graduates (Mbarika et al., 2010; Mbarika et al., 2003; Bagarukayo & Mbarika, 2008; Bradley et 
al., 2007a). It is therefore crucial to provide an education that encompasses HOCS to be 
successful in the work environment. 

Therefore there are several recommendations for revising the curriculum to emphasize on 
preparation of students to graduate with HOCS to enhance graduates' performance at the work 
place. New approaches to increase students' attention, interest, motivation, continuous learning 
desire and HOCS are continuously being sought. The multimedia (MM)-based form of instruction 
is one of the IT tools that have been developed to understand complex decision making and 
problem solving situations that require HOCS. Incorporating simulation or MM content into 
instructional materials has been argued to enhance student attention capturing, understanding, 
retention, engaging to use, and enhancing learning interest (Liu & Yuan, 2005; Jonassen, 1989). 
Studies based on student perceptions show that MM instructional materials improve HOCS 
(Mbarika et al., 2010; Bradley et al., 2007). However, there is insufficient research on the impact 
of MM technology on teaching, learning and assessment. 

In this paper we measure the impact that MM instructional materials have on 223 undergraduate 
students perceptions of HOCS improvement compared to traditional text book materials by 
investigating: - 

1] The existing effectiveness and impact of MM case studies on creative teaching strategies and 
perceived HOCS improvement. 
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2] The effectiveness of MM case studies as teaching / learning tools to enhance the emergence 
of best practices to guide successful implementation and foster perceived HOCS improvement. 

3] The study aims to evaluate the merits of MM to encourage HOCS development over traditional 
text book approaches. 

2. METHODOLOGY 

2.1 Participants 

A total of 497 second year Computer Science undergraduate students from a Ugandan university 
were contacted for the experiment. 223 students comprising of 70 females and 153 Males were 
selected to participate in the experiment because they were undertaking the operating systems 
course that semester, on which the case study is based. The selection for the control and 
experimental groups was random using the simple random sampling. 154 students in the 
experimental group accessed MM case study content online and 69 students of control group 
accessed the case using a text book for a period of one semester. Unfortunately the number of 
the students in MM and textbook groups is not the same because half of the students did not 
show up for the experiment. 

2.2 Data Collection 

2.2.1 Case study Content 

The chick-fil-A case study brings real-world problems from company into the classroom. The case 
study was developed with the objective of providing material so that theory, practice and design 
could be brought together to solve real life problems and provide material that improves HOCS 
(Mbarika ef a/., 2010; Sankar & Raju, 2000; Raju & Sankar, 2002; Mbarika, 2003). It illustrates the 
management decisions Chick-fil-A faced as the organization prepared to move from its current 
Point-of-sale (POS) system to a choice between two operating systems. The two POS systems 
are based on Windows NT and Windows CE technology. Students assumed the role of a MIS 
executive who had to choose between two operating systems for use as the company's POS 
system. The students worked in teams and each team had to determine which operating system 
is a better choice. The concepts to be covered included operating systems, business-technology 
alignment, joint application development process, and user interface issues. Since the Chick-fil-A 
chain operates over 700 corporately owned stores this changeover has an impact of 
approximately $3.29 million investment. Each team had to take the following into consideration as 
part of the decision making process (a) the mission of Chick-fil-A; (b) depreciation of existing 
systems; (c) projected return on investment of new systems; (d) total cost of ownership of existing 
and new systems; (e) employee retention, training, and education; (f) alignment of Chick-fil-A's IT 
Strategy and Business Strategy; (g) competitive advantage; and (h) Chick-fil-A’s critical success 
factors. In addition to this, a tutorial provided background information on Windows CE and NT. 
The team wrote a report and gave an oral presentation explaining their decision and 
recommendations. After the students completed the case study, they completed a questionnaire 
that measured perceived HOCS improvement. 

The openly available multimedia instructional materials used in the experiment were aligned to 
selecting between two POS Operating Systems for the simulation company Chick-fil-A's (Sankar 
& Raju, 2000). The MM content was available online for students to access freely using the 
usernames and passwords. The MM content had videos, images, audio and text and the text 
book content only features text and graphics. The text book was developed using the same case 
study content. The text book was printed and given to the students in the control group. Both the 
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text book and MM content is available online, so the students had access to it the entire 
semester. The experimental and control group were students from different classes doing the 
same course and hence there was no known interaction. The day students used MM and the 
evening students used the text book content. The students accessed the content during the class 
practical hours each week. The students were introduced to the case study during one lecture 
session. 

2.2.2 Experiment procedure 

Data was collected from students of both the control and experimental groups with the same case 
study content in text book and MM format respectively. The control group used the text book 
content of the same case, as explained in a text book, hence the traditional approach. The 
experimental group used the MM content of the Chick fil-A case study to evaluate HOCS 
improvement (Raju & Sankar, 2002; Mbarika, 2003). Both groups of students filled in online 
questionnaires which were analyzed to determine which of the instructional methods was better 
for perceived HOCS improvement. 

a) Attitude Pre-test and post-test questionnaire 

The researcher uses a modified multi-item questionnaire to measure the impact of the learning 
driven factors on the perceived HOCS construct. The multi-item questionnaire was developed to 
evaluate whether the case study (a) successfully brought real life problems to the classroom, b) 
was helpful in learning difficult management and computing topics, and c) was helpful in 
transferring theory to practice. The students filled in a pretest to determine their prior knowledge, 
perceptions and attitudes before the case study content was administered. The pre-test 
questionnaire was administered prior to intervention to collect background information related to 
knowledge of the topic under study, gender distribution, overall Grade Point Average (GPA), and 
related demographics. The questionnaire also deals with attitudes in several areas like general 
attitude toward subject matter, relevance to life and society, impact on cognitive domain of 
learning, impact on positive and negative aspects of affective domain, impact on teamwork, and 
communication skills. The pretest determined the students' attitudes to the subject by being 
administered to all subjects prior to the treatment to determine their prior knowledge. 

After the pretest questionnaire, the case study was administered by text book to the control group 
and MM online content to the experimental group for a period of one semester. They accessed 
the content in form of MM and text respectively, based on the group they belonged to. After the 
students had accessed the content, the post test questionnaire was administered to measure the 
change in HOCS and prior knowledge. The post-test questionnaires indicated how their learning 
improved in terms of the HOCS, and Learning-Driven constructs. It tested for attained knowledge 
after administering the instructional materials to measure treatment effects and the attitudes 
towards the material. It measured perceived HOCS improvement and the perceptions on the 
improvements achieved on the different items after the case study was administered. 

In the post-test phase, the questionnaire will test for the attained knowledge after administering 
the MM instructional materials to measure treatment effects and the attitudes towards the 
material. 

The items of the questionnaire are a result of progressive refinements and by their inclusion in 
multiple studies, possess a high degree of construct validity (Mbarika, 2003; Mbarika et al., 2010; 
Bradley et al., 2007a). The research used a modified multi-item questionnaire to measure the 
main constructs and items corresponding to learning-driven factors and HOCS factors and criteria 
to assess students' perceived learning. The questionnaire items measured the three Learning- 
driven constructs of learning interest, self reported learning, learned from others and one 
construct of HOCS improvement with items of problem solving, critical thinking, decision making, 
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and other skills to reinforce construct validity(Mbarika et ai., 2001; Mbarika et at., 2010; Hingorani 
eta!., 1998). 

The learning-driven factor and HOCS factor (Mbarika et at., 2001) are elaborated below and their 
items in table 1 

a) The Learning-driven factor explains how the instructional materials will be used as a 
tool to challenge the end-user in learning difficult IT and CS topics, in connecting theories 
and practice, in improving students’ understanding of basic concepts, and in providing the 
students a platform on which to learn from one another. 

b) The HOCS factor represents how an individual has acquired an adequate portfolio of 
skills that can be used to make decisions within a specified period of time. HOCS 
improvement was measured by a set of items that were validated in earlier research 
studies. 


Table 1: Constructs and items that are used to measure the factors in the research model 
(Mbarika et ai, 2001). 


Construct 

Definition 

Items 

Learning- 

Driven 

factors 

Composed of constructs that show 
the intrinsic value of the 
instructional materials to the end 

user. 

Constructs include self reported 
learning, learning interest, learned 
from others, and challenging 

Self 

Reported 
Learning 
(3 items) 

Measure of student's improvement of 
basic concepts and of identification of 
central management and technical 
issues through use of case study 

Improved my understanding of basic 
concepts, learned new concepts, 
learned to identify central management 
and technical issues. (Hingorani et ai, 
1998) 

Learning 
Interest 
(3 items) 

Measure of the level of student 
interest generated during and after 
the case study 

Discussed technical and managerial 
issues outside of class, did additional 
reading on technical and managerial 
issues, did some thinking for myself 
about technical and managerial issues. 
(Hingorani et ai., 1998) 

Learned 
from Others 
(2 items) 

Measure of how much the students 
learned from each other by valuing 
other student's point of view or 
interrelating important topics and 
ideas 

Learned to value other students' point 
of view, learned to inter- relate 
important topics and ideas. (Hingorani 
ef a/., 1998) 

HOCS- 

factors 

Improved ability to Identify, integrate, 
evaluate, interrelate concepts within 
the case study and make decision in a 
given problem solving situation 

Identify, integrate, evaluate, interrelate, 
problem solving, critical thinking, 
decision making 


Students were asked to indicate the extent of their agreement with the six evaluator statements in 
the questionnaire that measured the constructs on a 5-point Likert scale ranging from 1, strongly 
disagree, to 5, strongly agree. The items used to measure the perceived improvement of HOCS 
construct for the Chick-fil-A case include: 
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• I improved my ability to identify operating system issues. 

• I improved my ability to integrate operating system issues. 

• I improved my ability to evaluate critically operating system issues. 

• I became more confident in expressing my ideas. 

• I learned to interrelate important topics and ideas. 

• I learned to solve problems based on business theories. 


3. DATA ANALYSIS 

The researchers uploaded the instruments to a website for easy access, to ease the data 
collection and analysis process. The data was analyzed to determine the difference in the 
attitudes, perceptions and whether there was an improvement in perceived HOCS of problem 
solving, critical thinking, decision making and other skills. During the analysis phase the data 
collected was cleaned. Statistical Package for the Social Sciences (SPSS) was used to analyze 
data. The research questions were developed and investigated using the t-test since we were 
comparing two groups. The procedure solicited the perceptions of two groups of students, on the 
impact of learning driven constructs on perceived HOCS improvement when using different 
instruction methods. 

3.1 Measures 

The experiment explored the association between the instruction method and HOCS 
improvement while examining 4 dependent variables. The experimental design subjects were 
measured in terms of the dependent variables (pretesting), exposed to stimulus representing an 
independent variable (multimedia), and then remeasured in terms of the dependent variables 
(post testing). The attitudes attribute refers to the measure of the students change in attitudes 
after accessing MM and Textual content. Blooms Taxonomy highlights that the students learn 
when there is a change in attitudes (Bloom, 1956). We measure the change in the students' 
attitudes after accessing MM content as compared to the text book content by comparing the pre¬ 
test and post test questionnaires. The learning interest is the measure of the level of student 
interest generated during and after the case study (Mbarika et at., 2001; Hingorani ef a/., 1998). 
Self Reported Learning is the measure of student's perceived improvement of basic concepts of 
the course (Mbarika et al., 2001; Hingorani ef a/., 1998). It is the perceived ability to improve 
students understanding of basic concepts, new concepts, and identify central management and 
technical issues from the case study. The learned from others attribute is the measure of whether 
the students perceived they learned from each other by valuing other student's point of view or 
interrelating important topics and ideas (Mbarika et al., 2001; Hingorani et al., 1998). The attribute 
identifies the perception of students to learn from each other. 

As previously noted, perceived HOCS improvement in this study is the perception that an 
individual has acquired the required skills for decision making, problem solving and critical 
thinking, analysis, interpretation, among others (Mbarika et al., 2001). Decision making is the 
perceived ability for the student to make a selection and justify the choice. Problem solving is the 
perceived ability of a student to correctly find a correct solution to a problem. Critical Thinking is 
the perceived art of analyzing, synthesizing and evaluating thinking with a view of improving it 
(Paul & Elder 2009). It is the perceived ability to interpret, analyse, reason, analytically and 
reflectively think, draw conclusions and find solutions to problems, among others. We therefore 
measure whether there is a significant difference in the students' perception of their ability to think 
critically when they use MM and text book materials. 
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The ‘skills attribute' deals with the ability of students to attain general skills from the content. The 
skills in this case include writing, presentation skills among others. It is important to create a 
wholistic learner, who has attained knowledge, skills and attitudes (Bloom, 1956). 


4. RESULTS 

Demographic characteristics of all 223 students who participated are described in Table 2, 
stratified by the instrument. 


Table 2. Characteristics of the Sample (N-223) 


Variable 

Multimedia (N=154) 

Text (N=69) 

Gender 



Male 

66% (101) 

75% (52) 

Female 

34% (53) 

25% (17) 

GPA 



2.0-2.5 

7% (11) 

13% (9) 

2.51-3.0 

14% (21) 

19% (13) 

3.01-3.5 

20% (31) 

22% (15) 

3.51-4.0 

59% (91) 

46% (32) 

Experience 



Less than 1 year 

83% (127) 

74% (51) 

1 -2 years 

10% (15) 

16% (11) 

2-3 years 

6% (9) 

6% (4) 

More than 3 years 

2% (3) 

4% (3) 


Table 3. Descriptive Statistics of Factors 

Factors 

Group 

Obs. 

Mean 

Std. Dev. 

Attitude 

M 

154 

43.82 

9.77 


T 

69 

42.83 

11.18 

Learning Interest 

M 

154 

11.24 

4.26 


T 

69 

10.75 

3.76 

Self Reported Learning 

M 

154 

3.90 

1.63 


T 

69 

3.83 

1.58 

HOCS 

M 

154 

6.84 

2.74 


T 

69 

6.81 

2.45 

Critical Thinking 

M 

154 

11.04 

4.33 


T 

69 

10.68 

3.77 

Decision Making 

M 

154 

8.92 

3.61 


T 

69 

9.04 

3.45 

Problem Solving 

M 

154 

1.68 

1.46 


T 

69 

1.84 

1.39 

Learning from Others 

M 

154 

3.21 

1.38 


T 

69 

3.14 

1.30 

Skills 

M 

154 

12.91 

5.31 


T 

69 

12.44 

5.02 
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Students in MM group were of similar gender, GPA, and experience distribution to students in text 
group. In the MM group, a total of 154 students completed and returned the survey (Table 2). 
Over the half of these respondents (66%) were male, 59% of the sample had a higher than 3.5 
GPA. In both MM and text group, over half of these students had less than one year experience. 

A comparison of individuals with and without MM instruction by factors is presented in Table 3. An 
independent sample t-test was performed to examine the impact of exposure to technology on 
HOCS improvement. 

Results of an independent sample t-test presented in Table 4 indicate that students instructed 
with MM had higher means for all factors, except for “decision making” and “problem solving” than 
those with textbook instruction. However, these differences are statistically insignificant. 

Unexpectedly, both decision making and problem solving factors in MM group had lower mean 
scores (M=.92 and M=1.68, respectively) than those with textbook instruction (M=9.04 and 
M=1.84, respectively). However, these difference are statistically insignificant (t=.248 and t—.761, 
df=221, p>.05, respectively). 


Table 4. Results from Independent Sample t-test 


Factors 

Mean Difference 

t-statistic 

P-value 

Attitude 

.999 

.674 

.501 

Learning Interest 

.486 

.816 

.415 

Self Reported Learning 

.083 

.354 

.724 

HOCS 

.032 

.085 

.933 

Critical Thinking 

.357 

.593 

.554 

Decision Making 

-.127 

-.248 

.804 

Problem Solving 

-.158 

-.761 

.448 

Learning from Others 

.069 

.353 

.724 

Skills 

.466 

.617 

.538 


5. CONCLUSIONS AND FUTURE RESEARCH 

This study evaluates the impact of learning driven constructs on Perceived HOCS improvement 
when MM and text book materials are used by students from different backgrounds with different 
GPA, gender and work experience. We carried out an experiment on undergraduate students to 
determine the impact of learning driven constructs on the perceived HOCS improvement, when a 
case study was administered to two groups of students using the MM and text book content. 

We performed the independent t-test for comparing means between the two groups; MM 
and textbook. From the results we presented, there was insignificant difference between the 
treatment groups on students' perceptions on HOCS improvement and the learning-driven 
factors. The results indicate insignificant differences were observed between students who used 
the MM and text book case study. 

The negative results from this study agree with the findings of (Orr et al., 2001) who claim that 
MM-based information technology does not positively impact on learning. The similar attributes in 
MM and in textbook are the possible reason of the non significant differences findings. The non 
significant difference may be attributed to the fact that the materials were not appropriate to the 
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Ugandan context, since the case was based on a scenario from the United States. In future we 
will develop materials that represent the context they are familiar with. 

Therefore we conclude that the exposure to MM did not impact on all the constructs as compared 
to the textbook materials. The results also indicate that there was no difference in the students' 
attitudes, learning interest, learned from others; self reported learning, decision making, problem 
solving, and critical thinking. In conclusion therefore, based on the above results, exposure to MM 
during the learning process was not different form the text book materials significantly. Therefore 
there was no advantage (advancing their HOCS improvement) of using MM materials over 
traditional text materials. 

Our study dealt with only perceptual measures of HOCS improvement were measured, therefore, 
the researchers restricted the study to measuring students’ perceived HOCS improvement. There 
is difficulty in measuring actual HOCS improvement and a limited number of instruments available 
to measure actual HOCS improvement. Therefore the ability to measure actual HOCS 
improvement using multimedia instructional materials remains a challenge for future research. In 
future the experiment will be undertaken to measure whether actual HOCS improvement occurs 
when MM materials are used for instruction as compared to text. The researchers intend to 
measure whether actual HOCS improvement occurs by comparing students performance and 
GPA. The researchers will develop an instrument that will ask exam-like questions to determine 
whether the students actual HOCS improved after using MM materials. 

In future, we intend to find the specific technologies that can improve HOCS improvement, by 
testing different technologies that may impact on the different HOCS attributes of decision 
making, problem solving, critical thinking, analysis, interpretation etc. 
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